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Abstract 



Structural knowledge is a pre-requisite to valid performance assessment scores because structural 
knowledge leads to better transfer. Better transfer will cause consistency to occur between task performances for a 
given individual and this consistency allows for less construct-irrelevant variance and more construct-relevant 
variance to be present in the scores. This means more valid scores. 

This paper provides empirical support for each link in this argument. Concept maps are used as 
representations of the structural knowledge taught in a Political Science 1 course. Students in that course are 
assessed using an essay performance assessment. Those students in the concept-mapping condition are shown to 
write more consistent essays than those who do not use a concept map while writing. That is, we demonstrate that 
the amount of task-related variance is related to the immediacy of having constructed a concept map. Therefore 
structural knowledge is a pre-requisite to valid performance assessment 
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Structural Knowledge as a Pre-requisite to Valid Performance Assessment Scores 

* 

Since performance assessments tend to include only a small sampling of tasks, the weight of content 
sampling and construct representation is heavily borne by those few tasks. The ability to generalize from the 
performance on those few tasks to a broader domain hinges on the degree to which the examinee's performances on 
those two tasks are consistent. If the performances across tasks are highly consistent, generalizable inferences seem 
appropriate whereas if the performances are not consistent, then such generalizations are not appropriate. This 
generalizability is an issue of both reliability and validity. It is a reliability issue insofar as consistent performances 
along with congruent ratings will produce consistent, stable scores. It is a validity issue insofar as methodological 
concerns can be dismissed and construct issues considered, that is, there is more construct-relevant variance and less 
construct-irrelevant variance. At the moment, performance assessments routinely seem to deliver neither. 

The Problem of Task Related Variance in Performance Assessments 

A near consensus of opinion holds that reliable scores are difficult to get from performance assessments. 
Indeed, low reliability coefficients are seen as one of the major roadblocks, if not the pre-eminent roadblock, to the 
implementation of large-scale, high stakes performance assessments (Burger & Burger, 1994; Dunbar, Koretz, & 
Hoover, 1991; Herman, 1991; Linn, 1993, 1994; Linn, Baker, & Dunbar, 1991; Mehrens, 1992; Messick, 1994; 
Shavelson, Baxter & Pine, 1992). Low coefficients have been shown in numerous and diverse subject matters and 
populations such as behavioral observation in pre-school (McWilliam & Ware, 1994); middle school science 
(Shavelson & Baxter, 1992; Shavelson, Baxter & Gao, 1993); secondary school writing and mathematics (Koretz, 
Klein, McCaffrey, & Stecher, 1993; Koretz, Stecher, Klein & McCaffrey, 1994; Koretz, Stecher, Klein, McCaffrey 
& Deibert, 1993) and college writing (Nystrand, Cohen, & Dowling, 1993). 

The most problematic source of error variance - and hence, low reliability coefficients — is that of task and 
subject-by-task variance (Brennan, 1996; Brennan & Johnson, 1995; Green, 1995; Linn, 1993; Linn, 1994; Linn, 
Baker, & Dunbar, 1991; Swanson, Norman & Linn, 1995). In literally dozens of studies covering many diverse 
populations and subject areas, the lack of agreement between different tasks on an assessment has been documented 
(Baker, 1992; Breland, Camp, Jones, Morris, & Rock, 1987; Brennan, Gao & Colton, 1995; Coffman, 1966; College 
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Boards 1988; Dunbar, Korelz & Hoover, 1991 ; Gamache & Brennan, 1994; Gao & Colton, 1996; Gao, Shavelson, & 
Baxter, 1994; Ko^retz, Stecher, Klein, McCaffrey & Deibert, 1993; Lane, Liu, Ankenman, & Stone, 1996; Lane, 
Stone, Ankenman & Liu, 1994; Linn, 1993; Mehrens, 1992; Miller & Legg, 1993; Shavelson & Baxter, 1992; 
Shavelson, Baxter & Gao, 1993; Shavelson, Baxter & Pine, 1990; Shavelson, Baxter & Pine, 1992; Swanson, 

Norcini & Grosso, 1987). 

From the perspective of construct-relevant and construct-irrelevant variance (Messick, 1994, 1995), these 
task-related variances are also problematic. Variance associated with tasks is considered construct-irrelevant 
variance, and thereby reduces the validity of the scores from the assessment. 

Therefore, if performance assessments are ever going to be of use where high-quality scores are required, 
the issue of task-related variance needs to be addressed. That is, ways must be found to reduce task and subject-by- 
task variance components (Linn & Burton, 1994): But in order to do that, some explanation for task variance needs 
to be offered and some solution based on that explanation attempted. Since all assessments are fundamentally 
cognitive acts, or more specifically are acts of problem-solving (Snow & Lohman, 1989), peiiiaps an explanation 
lies in cognitive psychology. As Snow & Lohman (1989) state: . , new attempts to assess the more dynamic 

properties of task performance, understood in modem information-processing terms, might well pay off (p. 3 16).^' 
Task-Related Variance as a Transfer Problem 

Task variance and task-by-person variance are both problematic because, according to the generalizability 
sampling framework, they should be minimal. But they are quite common as has been previously cited. To put it 
another way, someone can be asked to do two or more different tasks and get two or more different results. This 
sounds very similar to research on the problem of transfer of learning. As Linn, Baker & Dunbar (1991 ) observe: 
^^The limited generalizability from task to task is consistent with research in learning and cognition that emphasizes 
the situation and context-specific nature of thinking (p. 19).'" It is in the transfer literature that a possible explanation 
for task-related variance can be found. 

The issues of transfer and analogical reasoning have vexed and perplexed educators and cognitive 
psychologists for a long time (cf. Lave, 1988). Students often can do a set of problems and seemingly understand a 
principle, yet they fail to employ that principle or solution in a novel situation. It seems reasonable to expect 
students to be able to do that, especially since transferring knowledge is an everyday part of life (Perkins & 
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Salomon, 1988), and yet much research exists to show that they often fail miserably at it. It sounds very similar to 
the problem of task-related variance: differential performance on tasks where none is anticipated. 

Analogical reasoning, one form of transfer, has long been the focus of much cognitive psychology research. 
It has been defined as “fundamentally the ability to utilize a well understood problem to provide insight and 
structure fora less understood problem (Grandgenett & Thompson, 1991; p. 294).” Gick& Holyoak(l983X who 
performed some of the classic work on analogical reasoning, define it thus: “The essence of analogical thinking is 
the transfer of knowledge from one situation to another by a process of mapping — finding a set of one-to-one 
correspondences (often incomplete) between aspects of one body of information and aspects of another (p. 2).” The 
conditions under which such a match is made are not clearly understood, but different suggestions have been made. 
Gick & Holyoak (1980) discuss the importance of the level of macrostructure at which this mapping takes place. 

That is, the mj^ping might be happening at a broad or general enough level as to be useless or, conversely, at a level 
too detailed to be useful. 

A typical transfer situation may consist of the following steps (Brown & Clement, 1989; Gick & Holyoak, 
1980; Novick, 1988; Perkins & Salomon, 1989; Pierce, Duncan, Gholson, Ray, & Kamhi, 1993). First, the base 
situation must be completely understood and its features and solution learned in generalizable terms. Second, the 
common elements between the base and the target must be noticed. Third, the relevant and useful aspects of the base 
must be extracted from memory. Fourth, the same relationships and solutions which worked in the base must be 
applied successfully to the target This process is further complicated, however, as the base and target become more 
and more dissimilar. In that case, several attributes in the base are irrelevant to the target and may actually be 
detrimental to solving the target situation (Novick, 1988). In this case, learners are presented vsdth the additional 
challenge of deciding which aspects of the base to use and which to ignore. 

Transfer is further complicated because the base and target, as well as the activity of transfer itself, are 
situated in some context which could very well alter the difficulty of making the transfer (Bassock & Holyoak, 

1989; Gick & Holyoak, 1980; Lave, 1988). Perkins & Salomon (1989) note that transfer is “highly specific and must 
be cued, primed, and guided (p. 19).” Gamer (1990) comments that little attention has been paid to this issue of 
context when addressing the issue of strategy use. 

In essence, analogical reasoning is an indeftnable and subtle balance between general principles or 
knowledge and specific situations. It is a balance because, if all of a learner’s knowledge is situationally specific, no 
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abstraction will take place and transfer will not occur (e.g. Bassock, 1990; Bassock & Holyoak, 1989; Gamer, 1990; 
Perkins & Salomon, 1989); conversely, if knowledge is very general, too few features exist with which to make 
matches, and again, transfer does not occur (Perkins & Salomon, 1989). Thus effective transfer or analogical 
reasoning is composed of both general and context-specific elements (Phye, 1989; Perkins & Salomon, 1989). The 
final complication to the process is that the learner must make the connection between the two situations somehow. 

Given these conditions, it seems surprising that transfer ever occurs. Indeed, it fails a great deal of the time 
and for various reasons. One possibility is that students fail to use consistent ^proaches to problems, perhaps due to 
inadequate learning of the base situation (Perkins & Salomon, 1988; Siegler, 1989). Ruiz-Primo, Baxter, & 
Shavelson (1993) report that students used different procedures to attack the same problems on different occasions. 
A second reason for failure to transfer depends on the type of problems being solved. Dl-defined problems — those 
with fuzzy parameters — provide more of a challenge to transfer than do well-defined problems because the mapping 
or matching process between the target and base is confounded. These types of problems are more common in the 
social sciences (Voss, 1988). A third type of transfer failure occurs when students are instructed only on general 
principles and see no concrete or context-specific examples of the principles in action (Bransford, Franks, Vye, & 
Sherwood, 1989; Bransford, Sherwood, Vye, & Rieser, 1986; De Leeuw, 1983; Larkin, 1989; Perkins & Solomon, 
1989; Stratton & Brown, 1972). 

These descriptions of transfer problems sound exactly like the problems encountered on assessments with 
large task and task-by-person variance. Green (1995) comments that on such assessments, the tasks have little 
common variance and much specific variance. This sounds much like Phye’s (1989) description of transfer 
consisting of both general and context-specific elements. To use the analogical reasoning vocabulary, the mapping 
of attributes between task variance situations and transfer failures seems to indicate that solutions to the base 
(transfer failures) might well be applicable to the target (task variance in performance assessments). Many different 
solutions to transfer problems have been attempted which focus on three basic parts of the transfer process. The first 
class focuses on students’ prior knowledge ( Bassock & Holyoak, 1989; Brown, Kane, & Echols, 1986; Gentner, 
1983; Gentner, 1989; Holyoak, 1984; Mayer & Bromage, 1980; Novick, 1988; Salomon & Perkins, 1989; Voss, 
1988; Wideman & Owston, 1991); the second class focuses on inducing a schema, that is, forming a generalizable 
knowledge base (Cooper & Sweller, 1987; Gick & Holyoak, 1983; Greeno, Moore, & Smith, 1993; Jelsma, Van 
MerriSnboer, & Bijlstra, 1990; Lambiotte & Dansereau, 1992; Novick, 1988; Phye, 1989; Royer, 1979; Royer, 
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1986; Rumelhart & Norman, 1980; Sweller, 1989; Van MerriOTboer & Paas, 1 990); and the third focuses on getting 
students to notice the correspondences between the base and the target (Catrambone & Holyoak, 1989; Gick & 
Holyoak, 1980, 1983; Phye, 1989; Ross, 1984; Ross, 1987; Salomon & Perkins, 1989; Voss, 1988). 

If the problem of task variance is actually a transfer problem, then these solutions should be applicable to it. 
Their implementation in performance assessment tasks should reduce task variance. Indeed, discussions of the task 
variance problem have included proposed solutions that sound very much like those proposed for transfer problems. 
Linn & Burton (1994) acknowledge that prior knowledge deficits might contribute to task variance, and if students 
have an opportunity to learn the content knowledge as well as gain practice with the task formats, task variance 
might be reduced. Suen, Sonak, Zimmaro, & Roberts (1997) have proposed a schema induction solution to task 
variance by hypothesizing that concept maps can be used to bridge the gap between tasks. As Linn & Burton (1994) 
observe, however, there is some theoretical work but little empirical work here. 

Bridging is a technique suggested to promote and teach transfer (Perkins & Salomon, 1988). The technique 
involves providing some support to help students see the possibility of transfer. Concept mapping is a powerful 
cognitive tool (Lambiotte & Dansereau, 1992; Novak & Gowin, 1984) and should provide an effective bridge across 
tasks for a number of reasons. First, research has shown that students with better schema are better able to transfer 
effectively (Catrambone & Holyoak, 1989; Cooper & Sweller, 1987; Gentner, 1989; Gick & Holyoak, 1983). More 
broadly, visual representations also aid transfer (Beveridge & Parkins, 1987; Gick & Holyoak, 1980; Gick, 1985; 
Yang & Wedman, 1993). The use of scaffolded techniques also aids transfer (Choi & Hannefin, 1995; Greenfield, 
1984; Harley, 1993; Paas, 1992). Finally, techniques which help students focus on the conceptual structure of 
knowledge are better at enhancing transfer than techniques that focus on surface features (Bransford, Sherwood, 

Vye, & Rieser, 1986; Gentner, 1989; Paas, 1992; Weaver & Kintsch, 1992). The present study is designed to benefit 
from all of these findings. 

This investigation brought all of this together empirically to address the issue of task-related variance. The 
review of literature suggests that task-related variance can be seen as a transfer problem and that solutions to transfer 
problems which draw on the structural knowledge technique of concept mapping might also address the task 
variance problem. The main hypothesis for this study is that the task-related variance components for those who 
constructed and used a concept map while writing will be smaller than the task-related variance components for 
those who did not construct or use a concept map while writing. 
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* Methodology 

This study took place in an introductory Political Science course in American National Government at a 
large, northeastern university. There were approximately 325 students enrolled in the course. The professor of the 
course proposes that politics can be thought of as a game with participants, rules, outcomes, resources, etc. It is 
from this underlying framework (the Play of Power) that the entire course is taught (Eisenstein, Kessler, Williams, 

& Switzer, 1996). That is, every aspect of American National Government covered in the course is explained using 
the metaphor. This provides the tight conceptual framework necessary to study issues of transfer. 

The primary analytical tool in the course was essay writing. Specifically, students were asked to take a 
newspaper article and understand and explain the situation in terms of the metaphor. Two of these essays were 
completed as part of an In-Class Graded Exercise (ICGE). In this study three of these ICGE’s are of interest. 

Four different treatment conditions were employed in this study. In the concept mapping condition, 
students were trained in the use of concept mapping as a way to organize the Play of Power metaphor. The 
‘^atmenf ' was that students were asked to create a concept map as a homework assignment and were allowed 
access to that concept map when working on the ICGE. During the ICGE, students who completed the map and 
brought it with them were told to refer to the map when formulating their responses. 

In the outlining condition, students were also trained to use outlining as a way to organize the course 
content As with the first condition, students were asked to produce an outline in advance to which they would be 
given access during the ICGE. The outlining condition here is a “placebo” control condition. That is, in order to rule 
out the competing hypothesis that concept mapping reduced variance simply because students had done more 
preparation for the ICGE's, this condition was added. The intention was that students would spend as much time 
creating the outlines, but the outlines would not provide the same benefits of enhancing schema formation, 
providing a visual representation, and improving conceptual understanding that should accrue through the use of 
concept maps. 

The third condition is the “no intervention,” or control, condition. Here, neither concept maps nor outlines 
were required or available to the students during the ICGE. 
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The fourth condition exists after ICGE 1 and is the previous concept map condition (PCM). These are 
students who had constructed and used a concept map on a previous ICGE but were not using one on the present 
ICGE. 

The twelve recitation sections of the course were broken into six experimental groups. The treatment 
conditions were then distributed across the four ICGE’s of interest to this study following a multiple baseline design 
so that, at each ICGE, some students were using a concept map and some were working unaided either as controls or 
PCM. 

The scoring of the ICGE’s was completed in three separate scoring sessions, one for each ICGE, The 
essays were scored for the ability of students to apply the concepts from the Play of Power to the article at hand. 
Specifically, raters judged the degree to which the student connects, integrates, elaborates and contextualizes the 
appropriate concepts. 

Not all students in each condition completed the homework assignment given to them, so the treatment 
groups were defined based on what was completed. Also, due to the cost of rating and the commitment of the raters, 
not every ICGE for every student was scored. Table 1 shows both which cells of the design were scored and what 
the effective cell sample size was. Since the outlining condition was primarily a form of control group whose 
functioning is reported elsev^ere (Parkes, 1998), those scores will not be of interest in this particular study. Only the 
Concept Map (CM), Control and PCM conditions are of interest for the first three ICGE’s. 



Insert Table 1 Here 



A separate generalizability study was conducted in each of the nine highli^ted cells in Table 1 following 
this model: = (7^ + + cr^ + O, +0"^ + . The task main effect variance components and the 

S t T St ST Tt STt 

subject-by-task variance components were all compared across different cells. A one-standard error confidence 
interval was constructed around each estimated component, and these confidence intervals were used to determine if 
the two components were of statistically significant different sizes. Also, the generalizability coefficients for each 
cell were calculated and compared. 
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Results 

Table 2 reports the estimated task main effect variance components and their associated standard errors. Table 3 
reports the estimated subject-by-task interaction effect variance components and their associated standard errors. 



Insert Table 2 Here 



Insert Table 3 Here 



The main hypothesis was tested through several different comparisons. Table 4 summarizes the comparisons made 
for the task main effect variance components. No statistically significant differences were observed for the estimated 
task main effect variance components. 



Insert Table 4 Here 



The comparisons for the estimated subject-by-task interaction variance components did produce some notable 
findings, as sunfimarized in Table 5. 



Insert Table 5 Here 



In two of the six comparisons of changes in the subject-by-task variance components, statistically significant 
differences were found. The concept mappers' subject-by-task variance component was smaller than the control 
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group’s. In the other four comparisons, statistical significance was not reached, though the relationships among the 
treatment groups trends according to the predictions made. 

From a practical standpoint, however, the real proof is in the effects these different variance components 
have on the generalizability coefficient With this particular variance model, the g-coefficient would be given by: 




£L- 

''2 ''2 



'' 2 






+ ■ 

n, n/i, 



where nr is the number of raters reading each essay (here, Hr =2); and nt is the number of tasks, or essays, 
written by each student (here, nt =2). 

In Figure 1 below, the g-coefficients for both the concept mapping condition and the control condition are 
given for each ICGE. Note that, for ICGE 3, the comparison is with previous concept mapping condition not the 
control condition. 



Insert Figure 1 Here 



In all cases, the g-coefficient is larger for the concept mapping group than for the control or previous 
concept mapping group. Thus the redistribution of variance due to concept mapping is having a positive effect. It is 
also noteworthy that the g-coefficients for the concept mapping condition continue to get larger throughout time 
whereas the control condition g-coefficients remain relatively flat across time. 

Conclusions 

These results suggest that students who have a concept map of the underlying conceptual framework at 
their disposal while they write produce more consistent applications of that framework than those students who do 
not. It is important to note that two effects did not occur. First, there was no effect on the task main effect variance 
components, only on the subject-by-task variance components. Second, there was no effect of having constructed a 
map previously but not accessing it while writing. 
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In retrospect, the lack of a main effect but the presence of an interaction effect is understandable. The effect 
of the concept map is an individual difference, not a blanket group effect. That is to &y that the quality and 
correctness of the maps and the ability of the student to employ the map while writing would both be different from 
student to student. That may be why no group effect was seen on the task main effect. As a group, the task variance 
was not different, but a smaller subject-by-task effect means that within each student, the task variance was less. So 
each students’ own two performances were more consistent. Therefore, it seems quite reasonable to have seen these 
effects. Having made those qualifications, some drop in task main effect theoretically should have been observed 
and it was not. 

The mixed results regarding the variance components from the Previous Concept Map condition highlight 
the issue of exactly what the concept maps are doing. There are two potential effects. First, the theoretical argument 
behind using concept maps was that the maps would strengthen and solidify the students’ internal cognitive network 
of political science concepts. The second effect is that having the concept map to refer to aided the transfer between 
the framework and the tasks but did not have a strong effect on the students’ internal cognitive structure. Since 
previous concept mappers did not perform as theory predicted when compared to control students, perhaps the 
second effect is the one observed here. Feedback from the teaching assistants with the course and other research 
being conducted on the maps themselves seem to indicate that the students did not fully implement concept maps. 
More specifically, some of the maps were quite poor; and some students expressed confusion about how the maps 
should help them. It seems more appropriate to make the argument that the maps were helpful “on the spot” but did 
not have lasting effect as they were implemented. This is not to say that the map was a “cheat sheet.” That 
conclusion is highly unlikely because the map did not contain “answers” or specific facts that would help with 
application of concepts. 

It is important to differentiate between the effects of consistent performances versus the effects of 
standardized tasks. The maps brought consistency to the students’ performances, not to the tasks themselves. The 
maps seem to have amplified the focus on the application of the underlying framework and de-emphasized the focus 
on the particulars of a given article as the students wrote. This, then, means that the shift in size of variance 
components and subsequent shifts in generalizability coefficients is more an issue of validity than reliability. That is 
to say that the maps did not standardize the task, otherwise, the main effect for task should also have been reduced. 
Rather the maps reduced the amount of construct-irrelevant variance and increased the amount of construct-relevant 
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variance in the generalizability model. This is shown in the generalizability coefficients shown in Figure 1. 
Therefore, there is stronger evidence of construct validity when students employ the concept maps while writing 
essays. 

Although these results are not strong, there is the suggestion that the line of logic used in designing the 
study seems to still be worth pursuing. There is ample literature and now some empirical findings to support each 
link in that chain. Structural knowledge, as represented by a concept map, leads to better transfer across tasks. That 
is, students with a concept map of the underlying framework do less task-specific thinking and more task-general or 
construct-specific thinking. If students are doing that, then their performances on multiple tasks should be more 
consistent. In other words, the subject-by-task variance components should be smaller. This in turn increases the 
proportion of construct-relevant variance and decreases the proportion of construct-irrelevant variance in the total 
score. And that means more valid scores. 
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Table 2 Estimated Task Main Effect Variance Components and Standard Errors 



Groups 


ICGE 1 


ICGE 2 


ICGE 3 


1 


Concept Maps 


Previous Concept Map 






0.023 


0 






(0.025) 


(0.016) 




2 




Concept Maps 


Previous Concept Map 






0.043 


0 






(0.042) 


(0.008) 


3 


Control 




Concept Maps 




0.007 




0 




(0.008) 




(0.019) 


4 


5 


6 




Control 








0 








(0.008) 
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Table 3 Estimated Subject-by>Task Interaction Effect Variance Components and Standard Errors 



Groups 


ICGE 1 


ICGE 2 


ICGE 3 


1 


Concept Maps 


Previous Concept Map 






0 


0.028 






(0.065) 


(0.049) 




2 




Concept Maps 


Previous Concept Map 






0 


0.141 






(0.051) 


(0.06) 
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Control 




Concept Maps 




0.177 




0.018 




(0.053) 




(0.071) 


4 


5 


6 




Control 








0.072 








(0.059) 
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Table 4: Summary of Task Main Effect Comparisons 





Comparison 


Result 


ICGEl Concept Map Condition 


ICGEl Control Condition 


n.s. 


ICGE2 Concept Map Condition 


ICGE2 Control Condition 


n.s. 


ICGE2 Concept Map Condition 


ICGE2 Previous Concept Map Condition 


n.s. 


ICGE2 devious Concept Map Condition 


ICGE2 Control Condition 


n.s. 


ICGE3 Concept Map Condition 


ICGE3 Previous Concept Map Condition 


n.s. 


ICGEl Control Condition 


ICGE3 Concept Map Condition 


n.s. 



25 




25 



Table 5: Summary of Subject-by-Task Interaction Effect Comparisons 





Comparison 


Result 


ICGEl Concept Map Condition 


ICGEl Control Condition 


CM < Control 


ICGE2 Concept Map Condition 


ICGE2 Control Condition 


n.s.*** 


ICGE2 Concept Map Condition 


ICGE2 Previous Concept Map Condition 


n.s.”** 


ICGE2 Eh-evious Concept Map Condition 


ICGE2 Control Condition 


n.s.”** 


ICGE3 Concept Map Condition 


ICGE3 Previous Concept Map Condition 


n.s.’*' 


ICGEl Control Condition 


ICGE3 Concept Map Condition 


CM < Control 



♦ These comparisons, though not statistically significant, do trend according to the prediction made in the 
hypothesis. 
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